Batch-scheduling Data Flow Graphs with Service-level Objectives on Multicore Systems

Data flow graphs are a popular program representation in machine learning, big data analytics, signal processing, and, increasingly, networking, where graph nodes correspond to processing primitives and edges describe control flow. To improve CPU cache locality exploit data-level parallelism, usually process batches. Batchy is scheduler for based packet engines, which uses controlled queuing reconstruct fragmented batches inside accordance with strict Service-Level Objectives (SLOs). Earlier work showed that yields up 10x performance improvement real-life use cases, thanks maximally exploiting batch gains. Batchy, however, fundamentally restricted single-threaded execution. In this paper, we generalize parallel execution on multiple cores. We extend the analytical model setting present primal decomposition framework, each core runs an unmodified controller schedule batch-processing subset of graph, orchestrated by master distributes delay-SLOs across cores using subgradient search. Evaluations real software switch provide experimental evidence our framework produces 2.5x while accurately satisfying delay SLOs otherwise not feasible single-core Batchy.